Conversation
…g and centralized state management
There was a problem hiding this comment.
Pull request overview
Implements a workspace-scoped, multi-file upload pipeline that persists per-file schema metadata, infers join candidates server-side, and surfaces that context to the frontend via centralized state. In parallel, it removes the Eval dashboard surface area and includes several backend reliability/security improvements (MIME sniffing, parse warnings, disconnect handling, lifespan + eviction supervision).
Changes:
- Added workspace-aware upload + file listing/deletion endpoints backed by a new
workspace_filestable, plus aschema_mergerservice andMultiFileContextmodel for join-candidate inference. - Updated the React upload flow/store to use workspace-scoped endpoints, track file-loading state, and store
multiFileContext+ join overrides. - Removed Eval dashboard codepaths (frontend + backend) and tightened backend orchestration behavior (token budget guard, supervised tasks, async JWKS fetch, parse warnings).
Reviewed changes
Copilot reviewed 77 out of 77 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/stores/workspaceStore.js | Adds multi-file context + join override state/actions to the workspace store. |
| src/services/evalApi.js | Removes eval dashboard API client. |
| src/services/api.js | Routes uploads to workspace-scoped endpoint when workspaceId is present. |
| src/pages/ProjectsPage.jsx | UI/layout tweaks (pagination, footer, scroll behavior) and removes unused auth token accessor. |
| src/pages/HomePage.jsx | Wires filesLoading into upload panel; removes eval link/metrics props. |
| src/hooks/useFileUpload.js | Fetches workspace files from new endpoint; maps schema_json dtypes; stores multi_file_context from upload responses; adds filesLoading. |
| src/hooks/useAgentRun.js | Removes metrics state; switches DEFAULT_SETTINGS import to constants module. |
| src/contexts/AuthContext.jsx | Disables react-refresh export rule (lint workaround). |
| src/components/upload/FileUploadPanel.jsx | Adds loading empty-state (spinner) when files are being fetched. |
| src/components/shared/WorkspaceSelector.jsx | Simplifies create handler (drops useCallback). |
| src/components/OutputPanel.jsx | Adjusts bullet-cleaning regex. |
| src/components/eval/TracePanel.jsx | Removes eval trace UI. |
| src/components/eval/RunList.jsx | Removes eval run list UI. |
| src/components/eval/KpiCard.jsx | Removes eval KPI card UI. |
| src/components/eval/DebugChart.jsx | Removes eval debug chart UI. |
| src/components/eval/AgentTable.jsx | Removes eval agent table UI. |
| src/components/agent/HistoryPanel.jsx | Removes eval-metrics display from history items. |
| src/components/agent/CodeBlock.jsx | Changes CodeBlock props (drops onToggleCollapse). |
| src/components/agent/agentSettingsConstants.js | New constants module for DEFAULT_SETTINGS (Fast Refresh compatibility). |
| src/components/agent/AgentSettings.jsx | Imports DEFAULT_SETTINGS from new constants module. |
| src/components/agent/AgentProgressPanel.jsx | Removes metrics overlay usage and tweaks bullet-cleaning regex. |
| src/components/agent/AgentMetricsOverlay.jsx | Removes metrics overlay component. |
| src/App.jsx | Removes /eval route; minor hook usage adjustment. |
| scratch/test_supabase_schema.py | Removes scratch script. |
| scratch/test_planner.py | Removes scratch script. |
| scratch/test_planner_debug.py | Removes scratch script. |
| scratch/test_env.py | Removes scratch script. |
| scratch/test_dump.py | Removes scratch script. |
| scratch/test_db_data.py | Removes scratch script. |
| scratch/backfill_user_id.py | Removes scratch script. |
| README.md | Updates tech stack/docs and removes eval/API reference sections. |
| eslint.config.js | Expands unused-vars ignore patterns; adds Node globals for config files. |
| backend/utils/helpers.py | Removes unused UUID ID generator helper. |
| backend/tests/test_upload_controller.py | Adds controller tests for workspace upload/delete + legacy upload behavior. |
| backend/tests/test_schema_merger.py | Adds tests for join inference + prompt/header generation. |
| backend/services/upload_service.py | Adds magic-byte MIME validation and cache locking for thread safety. |
| backend/services/supabase_service.py | Fixes file listing scoping by user; adds workspace_files CRUD; updates timestamps to UTC-aware; removes eval-metrics persistence. |
| backend/services/schema_merger.py | New join-candidate inference service producing MultiFileContext. |
| backend/services/process_service.py | Captures near-empty parse warnings separately from hard failures. |
| backend/services/parsers/txt_parser.py | Raises ParseWarning for near-empty parsed text; improves decode/sanitize error handling. |
| backend/services/parsers/pdf_parser.py | Raises ParseWarning for near-empty extracted text; improves error specificity. |
| backend/services/parsers/parquet_parser.py | Uses JSON-safe record sanitization for sample rows. |
| backend/services/parsers/md_parser.py | Raises ParseWarning for near-empty parsed text; improves decode/parse error handling. |
| backend/services/parsers/excel_parser.py | Uses JSON-safe record sanitization for sample rows/previews. |
| backend/services/parsers/csv_parser.py | Uses JSON-safe record sanitization for sample rows/sample_data. |
| backend/services/parsers/init.py | Introduces ParseWarning and sanitize_records helper. |
| backend/requirements.txt | Adds kaleido for plotly static export. |
| backend/models/multi_file_context.py | Adds canonical Pydantic models and prompt/reader-header helpers for multi-file context. |
| backend/models/metrics_schema.py | Removes DS-STAR metrics schema (eval/metrics de-scoped). |
| backend/middleware/auth.py | Makes JWKS refresh async + cached with rotation retry; updates JWT decode flow. |
| backend/main.py | Migrates to lifespan; adds supervised session-eviction background task; centralizes config health logging. |
| backend/eval/schemas.py | Removes eval schema module. |
| backend/eval/eval_store.py | Removes eval persistence store. |
| backend/eval/eval_routes.py | Removes eval API routes. |
| backend/eval/eval_logger.py | Removes eval SSE sidecar logger. |
| backend/eval/eval_engine.py | Removes eval metrics computation engine. |
| backend/eval/init.py | Removes eval package marker. |
| backend/db/migrations/004_workspace_files_file_size.sql | Adds file_size column and widens file_type constraint (txt/pdf/etc). |
| backend/db/migrations/003_workspace_files.sql | Adds workspace_files table + RLS + index + backfill. |
| backend/db/create_eval_schema.sql | Removes eval schema SQL. |
| backend/db/add_eval_metrics_fk.sql | Removes eval FK migration SQL. |
| backend/core/planner/planner_agent.py | Adds step_type and join-hints injection; wraps LLM calls with timeouts. |
| backend/core/ds_star_orchestrator.py | Adds context budget guard + per-call max_rounds; removes eval metrics event emission. |
| backend/core/deep_research_orchestrator.py | Uses fresh orchestrator per subquestion (state isolation); forwards per-call max_rounds; improves event timing. |
| backend/core/debugger/debugger_agent.py | Injects multi-file join context into debugger on merge-related errors. |
| backend/core/config.py | Adds sandbox RLIMITs and context budget reserve; removes unused IDP formats constant. |
| backend/core/analyzer/file_analyzer.py | Adds analyze_multi and disk fallback for file content; multi-file prompt header. |
| backend/api/routes.py | Adds typed create-workspace request; adds workspace upload/list/delete endpoints; passes user_id to file list; removes eval router. |
| backend/api/controllers/research_controller.py | Adds disconnect monitoring; forwards max_rounds; emits parse warnings; workspace auto-hydration. |
| backend/api/controllers/process_controller.py | Adds get_schema_context helper for workspace multi-file schema context. |
| backend/api/controllers/agent_controller.py | Adds workspace auto-hydration; forwards per-call max_rounds/workspace_id; emits parse warnings; removes eval logger. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+39
to
+53
| # Allowlist: extension → set of acceptable MIME types returned by libmagic. | ||
| # Only file types the platform actually needs to process are permitted. | ||
| _ALLOWED_MIME_BY_EXT: Dict[str, set] = { | ||
| "csv": {"text/plain", "text/csv", "application/csv"}, | ||
| "tsv": {"text/plain", "text/tab-separated-values"}, | ||
| "txt": {"text/plain"}, | ||
| "json": {"application/json", "text/plain"}, | ||
| "xlsx": { | ||
| "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", | ||
| "application/zip", # libmagic sees XLSX as zip | ||
| }, | ||
| "xls": {"application/vnd.ms-excel"}, | ||
| "parquet": {"application/octet-stream"}, | ||
| "pdf": {"application/pdf"}, | ||
| } |
| import { Copy, Check } from 'lucide-react' | ||
|
|
||
| export function CodeBlock({ codeByTab, collapsed, onToggleCollapse }) { | ||
| export function CodeBlock({ codeByTab, collapsed }) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…g and centralized state management